Introduction

Purpose

The purpose of the present document is to provide an installation guide and to describe the mode of operation of the Auxiliary Data Conversion System Next Generation (ADCSng) and a summary of its main functionalities. After reading this document the user should be able to install, work with and fully understand the software package.

Running ADCSng in a nutshell

  1. Obtain ADCSng from BitBucket: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/adcsng/

  2. Install from top level directory with "pip3 install -e ."

  3. Generate an ADCSng configuration file as from adcsng/adcsng/tests/bc/bepicolombo.json

  4. Generate the appropriate directory structure as derived from the configuration file

  5. Obtain and install your SPICE Kernel Dataset: https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS

  6. Run with "adcsng yourconfig.json"

Overview

Auxiliary data are those that help scientists and engineers to determine the location and orientation of the spacecraft, when and how an instrument was acquiring scientific data. These data also help to determine what other relevant events were occurring on the spacecraft or ground that might affect the interpretation of the scientific observation or the S/C systems performance. Software applications are required to know what were the location, size, shape and orientation of the observed target in addition to these auxiliary data. Almost all NASA and ESA planetary missions have embraced the use of the SPICE system for ancillary data archiving and for science data analysis. Although the Flight Dynamics Division provides software to read the position and orientation files of the orbiters, most of the Principal Investigators (PI) have pointed out their interest in having all the auxiliary data distributed in SPICE format.

The ESA SPICE Service (ESS) of the Cross Mission Support Office is responsible for supporting the ESOC auxiliary data transformation into SPICE kernels and the distribution of these files to the instrument teams using the public network. NASA’s Planetary Data System discipline node NAIF (Navigational Ancillary Information Facility) is responsible for designing a transformation software tool to generate spacecraft orbit and attitude SPICE kernels. The ESS, helped by the individual instrument and FDy teams, the SGSs, and, in some missions, supported by NAIF, design and support all the necessary instrument, spacecraft and timing kernels.

Applicable and Reference Documents

ADCSng in based on different technologies, in particular most of the code is written in Python. FORTRAN is also used along with other technologies such as JSON, being familiar with those technology will certainly facilitate the usage of ADCSng.

Applicable Documents

ID

Title

Reference

AD-E40

ECSS-E40 Space Software Engineering

ECSS-E-ST-40C

AD-PYP

Packaging and Distributing Projects

https://packaging.python.org/tutorials/distributing-packages/

AD-JSN

Introducing JSON

http://www.json.org/

AD-BBT

Bitbucket

https://bitbucket.org/product

Reference Documents

ID

Title

Reference

RD-ADC

Auxiliary Data Conversion into SPICE Kernel And Distribution

MEX-EST-PL-10210

RD-ANG

Auxiliary Data Conversion into SPICE Kernels for ESA Solar System Exploration Missions

SPICE-CROSS-TN-001

RD-ECM

ESA SPICE Service Configuration Management Plan

SPICE-CROSS-TN-002

RD-PIP

Configuring a .pypirc File for Easier Python Packaging

https://truveris.github.io/articles/configuring-pypirc/

Terms, Definitions and Abbreviated Terms

Acronym

Name

SPICE

Orbit Ephemeris Message

SPK

Attitude Ephemeris Message

CK

Time Correlation Packets

SCLK

JavaScript Object Notation

ORBNUM

ExoMars2016

MK

BepiColombo

OEM

Solar Orbiter

AEM

Jupiter Icy Moons Explorer

HK TM

Mars Express

FDy

Flight Dynamics

SGS

Science Ground Segment

Software Overview

ADCSng together with some other scripts and pieces of software cover from the automated download of orbital, attitude and spacecraft (S/C) and Payload elements position and orientation data from the given mission Science Ground Segment (SGS) uplink and downlink systems as provided by FDy in the shape of Orbit Ephemeris Messages (OEM), Attitude Ephemeris Messages (AEM) -or another Orbit and Attitude information container files- and Raw Housekeeping Telemetry data processed following the NASA Planetary Data System Standard (PDS4). More information on its concept and design is available here: [RD-ANG]

Installation

Hardware and Software requirements

ADCSng is a Python 3.6.x package that uses FORTRAN 77 and FORTRAN 90 applications and a set of standard Python libraries. ADCSng is meant to interact with different nodes: SGS Uplink and Downlink sub-systems, the ESA SPICE FTP Server and the NAIF FTP Server.

ADCsng is designed to work with UNIX systems namely with Linux and with MacOS. The hardware requirements are minimal and any modern computer/server will suffice. The software is able to run for example in a minimal CentOS 7 Virtual Machine:

  • Architecture: x86_64 (64 bits)

  • Memory: 1GB/logical CPU

  • Disk space: 10GB

The software requirements are the following:

  • Python 3.6 or higher

  • Python packages:

    • numpy >= 1.8.0

    • pytest >= 2.9.0

    • pandas > 0.17.1

    • html-testRunner

  • gcc

  • libgfortran

  • Git (possibly, for the obtention of SPICE Kernel Datasets)

Please note that the FORTRAN applications are already compiled and therefore the user should not compile and link them. Nevertheless this is possible since the source code is distributed with ADCSng, the only remaining part would be to link them with the SPICELIB library of the SPICE Toolkit. The compilation of these applications is out of the scope of this SUM and any interested user is encourage to try by himself or to contact the author.

Obtaining the Software

Installing from the ESS Nexus repository with PiPy

ADCSng can be installed directly from PiPy without the need to download the source code, in order to do so, you need to configure the given PIP installation by editing your local Python Package Index configuration file. More information in RD-PIP.

The file is accessible with: more ~/.pypirc what follows is the current configuration for ADCSng:

[distutils]
index-servers = nexus-master nexus-develop
 
[nexus-master]
repository = https://bepicolombo.esac.esa.int/nexus/repository/spice-pip-release/
username = spice-deploy
password = esacvilspa1
 
[nexus-develop]
repository = https://bepicolombo.esac.esa.int/nexus/repository/spice-pip-develop/
username = spice-deploy
password = esacvilspa1

Installing the source code

The source code of ADCSng is under configuration control with Git the BitBucket repository that hosts it is the following: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/adcsng/browse, since this is a private repository access might not be granted, if so please contact the author of this document. Alternatively the source code can be obtained under request from esa_spice@sciops.esa.int.

If the user is to install ADCSng from its source code it is recommended to use PiPy to do so (taking advantage of the Python setup file), in order to do so using the Terminal go to the top level adcsng directory, adcsng and type:

pip3 install [--user] -e .

Please note that the --user parameter is optional; if you are using your user Python installation it needs to be indicated.

Once installed, you will be able to run ADCSng anywhere in your computer.

Source Code structure

A brief overview of the code structure for ADCSng is useful for its installation, configuration and usage:

ADCSng Package top level structure

LICENSE

 

License file (as per any other Python package).

MANIFEST.in

Manifest file (as per any other Python package).

README.md

Readme file (as per any other Python package).

setup.cfg

Configuration for Python Setup.

setup.py

Setup Python routine.

adcsng

Package directory.

 

command_line.py

 

Main function called when invoking ADCSng.

 

classes

Directory that contains Python source code classes.

 

config

Directory that contains the mission-specific developer configuration items for ADCSng.

 

core

Directory that contains Python source code core functions.

 

etc

Directory that contains different templates used by ADCSng (mostly the ones used by the FORTRAN utilities).

 

exe

Directory that contains Linux and MacOS FORTRAN utilities executables.

 

src

Directory that contains the FORTRAN utilities source code.

 

utils

Directory that contains Python source code utility functions.

 

tests

Directory that contains the tests performed for ADCSng.

 

 

bc

 

Directory that contains the test files for BepiColombo (shown as example).

 

 

 

bepicolombo.json

User configuration file for BepiColombo testes (shown as example).

Configuration

The next step to use ADCSng is to configure it. ADCSng has two levels of configuration:

  • User configuration

  • Developer configuration (placed under the adcsng/adcsng/config directory)

As a user you only need to take care of the user configuration. the user configuration has to be edited and prepared by the user. It consists of a JSON file and an example from the test directory of ADCSng can be used as baseline (adcsng/adcsng/tests/bc/bepicolombo.json). Any name can be provided to the configuration file, the file extension has to be *.json. The configuration file will determine the directory structure that you need to setup for ADCSng. The recommendation is to create a directory out of the ADCSng directory structure and to create one per intended ADCSng execution/project.

The following table describes all the available configuration items (please note that all the items are strings):

Key

Value(s)

Description

Required?

name

ADCSng Configuration File

Description of the JSON file

NO

mission

  • BEPICOLOMBO

  • ExoMars2016

  • ExoMarsRSP

  • JUICE

  • MARS_EXPRESS

  • VENUS_EXPRESS

  • Solar_Orbiter

Name of the mission, has to be one of the provided in the list. (corresponds to the name of the SKDs replacing the dashes).

YES

operational_host

It is recommended to obtain the operational host by running the hostname command in your terminal.

Name of the operational Host.

If you use multiple environment and wish to distinguish in between test/development environments from the operational environment, this will allow you to receive notifications with an indication of whether if the run has been done with the operational pipeline or not.

YES

source_dir

Directory path for the input files.

Directory path for the input files. This directory is the one that ADCSng will look for input.

YES

temp_dir

Directory path for ADCSng temporary directory.

Directory path for the where ADCSng will place temporary files.

WARNING: DO NOT PUT ANY FILE IN THIS DIRECTORY: It will be deleted by ADCsng.

YES

log_dir

Directory path for ADSCng log files.

More information on log files below.

YES

kernels_dir

Directory path for the SPICE Kernel Dataset of the mission to be executed.

This has to point to the kernels directory of the SDK. More information below.

YES

output_dir

Directory path for the output of ADCSng.

Directory where the output will be placed.

WARNING: It is HIGHLY RECOMMENDED that this path is the same as kernels_dir.

More information below.

YES

processed_dir

Directory path to place the used and processed input.

More information below.

OPTIONAL

log_rotation

  • Daily

  • Weekly

  • Monthly

This key indicates the log rotation. The log will be split on a daily, weekly or monthly basis. More information below. If not indicated the default will be Monthly.

OPTIONAL

daily_log

Directory path to place an extra log generated per pipeline execution.

More information below.

OPTIONAL

lock_file

Path and name of the lock file.

More information below.

OPTIONAL

mkgen

  • True

  • False

  • Only

Parameter that indicates whether if ADCSng generates a meta-kernel for the run or not or if it is only executed to generate a meta-kernel (and ignore the input).

YES

email/send_email

  • True

  • False

Parameter that indicates whether if e-mails will be sent for clients after an execution and for the developer for logs and for failed executions. More information below.

YES

email/clients

  • email address for clients

  • Blank (if send_email false)

E-mail list for clients to receive notifications of the pipeline execution. More info below.

YES

email/developer

  • email address for developers

  • Blank (if send_email false)

E-mail list for developers to receive notifications of the pipeline execution. More info below.

YES

email/git_repository

HTTP(S) of the Git repository that contains the kernels

Free text to introduce the HTTP(S) address of the Git repository for the SKD.

YES

email/ftp_address

HTTP(S) of the FTP repository that contains the kernels

Free text to introduce the HTTP(S) address of the FTP repository for the SKD.

YES

email/skd_name

Name of the Git repository for the given mission.

Free text to introduce the name of the SPICE Kernel Dataset as in: https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS

YES

email/mission

Name of the FTP directory for the given mission.

Free text to introduce the name of the SPICE Kernel Dataset as in: ftp://spiftp.esac.esa.int/data/SPICE/

YES

email/contact

Contact name and/or email

Free text to introduce the Contact information for clients.

YES

Once the configuration file has been generated it can be placed anywhere in your computer, it is recommended to put it in the same directory level as where you will define the directory structure of your ADCSng configuration. What follows is an example of an ADCSng configuration file:

{
"name": "ADCSng Configuration File",
"mission": "BEPICOLOMBO",
"operational_host": "ESAC1102875",
"source_dir": "/Users/esaspice/bc/incoming/",
"temp_dir": "/Users/esaspice/bc/temp",
"log_dir": "/Users/esaspice/bc/log",
"lock_file": "/Users/esaspice/bc/adcsng.lock",
"kernels_dir": "/Users/esaspice/SPICE/BEPICOLOMBO/kernels",
"output_dir": "/Users/esaspice/SPICE/BEPICOLOMBO/kernels",
"processed_dir": "/Users/esaspice/ANCDR/BEPICOLOMBO",
"log_rotation": "daily",
"daily_log": "True",
"mkgen": "True",
"email": [{
"send_email": "True",
"clients": "mcosta@sciops.esa.int",
"developer": "mcosta@sciops.esa.int",
"mission" : "BepiColombo",
"skd_name" : "BEPICOLOMBO",
"ftp_address": "ftp://spiftp.esac.esa.int/data/SPICE/",
"git_repository": "https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS/repos/",
"contact": " Marc Costa\n mcosta@sciops.esa.int"
}]
}

If we consider the previous configuration file we could have a directory structure as follows:

bc
|-- adcsng.json
|-- incoming
|-- kernels -> ../ftp/data/SPICE/BEPICOLOMBO/kernels/
|-- log
|   |-- adcsng_weekly_2019Oct04.log
|   '-- adcsng_weekly_2019Oct09.log
|-- misc -> ../ftp/data/SPICE/BEPICOLOMBO/misc/
|-- processed -> ../ftp/data/ANCDR/BEPICOLOMBO/
'-- temp

Remember that you need to generate this directory structure for ADCSng will not generate it for you.

Kernels and Output directories requirements

Regardless of the kernels/output directory structure you setup the following subdirectories for each path are mandatory in order to execute ADCSng:

  • kernels: 'ck', 'spk', 'lsk', 'sclk', 'pck', 'ik', 'fk' and 'mk'

  • output: 'ck', 'spk', 'sclk', 'mk'

If any of those kernels sub-directories is not present ADCSng will trigger an error and the execution will fail.

You need a working SPICE Kernel Dataset to run ADCSng

It is important to note that you need a local copy of a working SKD in order to execute ADCSng. This document does not intend to provide you a detailed explanation on how to obtain the latest version of a given SKD, instead we recommend you to read the information in the following page: https://www.cosmos.esa.int/web/spice/data

What is important to note though is that you will need to have a working SKD. In order to check that you do we recommend you to use a SPICE utility such as BRIEF on the MK in order to ensure that the SKD is properly installed.

ADCSng needs the SKD as an input and preferably also output directory for the execution: the idea is that ADCSng updates the SKD itself and therefore it should be considered as both an input and an output. That is why in general is recommended to set the kernel and the output directories with the same value.

If you still consider to have different directories please note that the SCLK that you generate with an ADCSng run will not be taken into consideration to generate the rest of the CK kernels; instead the SCLK in the kernel directory will be used. The same issue will happen with the versioning of the kernels you generate: if the same kernel is generated the version will not be derived from the original SKD.

Processed directory requirements

The processed directory should have a given sub-directory structure. This structure is determined by the 'originators' of each input file. ADCSng will generate these sub-directories if they do not exist. The 'originator' acronyms can be found in the developer configuration. In general they are the following:

  • man: Files coming from Mission Analysis (typically test trajectories)

  • fdy: Files coming from Flight Dynamics (typically OEMs and AEMs)

  • soc: Files coming from the Science Operations Center (typically planning AEMs)

  • hkt: Files from the Housekeeping Telemetry typically processed by the SOC Downlink group

  • tcp: Time Correlation Packets coming from the S/C

Please note that the files under these directories are usually published in the ESA SPICE FTP. An example can be seen in the following link: ftp://spiftp.esac.esa.int/data/ANCDR/BEPICOLOMBO/

If the 'processed_dir' is not specified in the condiguration file (is left empty) then the processed files will simply not be copied (this is the case for ExoMarsRSP for instance).

Executing ADCSng

In order to execute ADCSng you only need to type the command: adcsng, the command usage is as follows:

adcsng [-h] [-v] [-m] [-l] [-d] CONFIG [CONFIG ...]

Indicating the configuration file path is mandatory, the options are displayed by the -h or --help options:

___ ___ _________ Auxiliary Data Conversion System next generation
/ _ | / _ \/ ___/ __/__ ___ _
/ __ |/ // / /___\ \/ _ \/ _ `/ a command-line utility program that
/_/ |_/____/\___/___/_//_/\_, / converts auxiliary data into SPICE
/___/ Kernels for the European Space Agency
v1.1.0
 
positional arguments:
CONFIG Mission specific JSON configuration file
 
optional arguments:
-h, --help show this help message and exit
-v, --version Display the version of ADCSng
-m, --metakernel Execute ADCSng to generate meta-kernel only
-l, --log Prompt log during execution
-d, --debug Run in debug mode; run files not cleaned-up, etc.
 
Source and documentation is available here:
https://repos.cosmos.esa.int/socci/projects/SPICE/repos/adcsng
 
developed and maintained by the
__ __ __ __ __ __ ___ __ ___ __ __ ___
/__\ /__` '__\ /__` |__) | / ` |__ /__` |__ |__) \ / | / ` |__
\__, .__/ \__/ .__/ | | \__, |___ .__/ |___ | \ \/ | \__, |___
 
esa_spice@sciops.esa.int for Mars-Express // ExoMars2016 // ExoMarsRSP
http://spice.esac.esa.int BepiColombo // Solar Orbiter

Please note that if you do not use the optional argument -l or --log you will not be prompted the log of the execution. Error messages prior to the log creation might be shown though.

Once ADCSng is executed, in a nutshell it checks the incoming directory for input data. If present it processes the input data and generates the corresponding kernels (if no input data is present but ADCSng is set to generate a meta-kernel it generates it) and then moves the generated kernels to the output directory and the input and processed data to the processed directory. Finally if specified ADCSng notifies the users and developers with an e-mail.

Please note that the synchronisation with BitBucket and other services such as the SKD validation are performed by other packages of the ESA SPICE Service (such as spigit and spival).

Logging

ADCSng provides multiple options for the logging process, the logging is provided via:

  1. Log file with a specified rotation written in a specified directory. Filename is adcsng_rotation_YYYYMonDD.log where:

    1. rotation is 'monthly', 'weekly' or daily,

    2. YYYY is the year, Mon is the three letter capitalised month acronym and DD is the day with two digits.

  2. Log file per execution written in a specified directory. Filename is adcsng_YYYYMMDDTHHMMSS.eop where:

    1. YYYY is year, MM month, DD day, HH hour, MM minute, SS second

    2. eop stands for End of Process.

  3. E-mail containing the log file text per execution

  4. Prompted Log in the screen if -l or --log optional argument is provided

Log format

The log format per line is as follows (Python syntax):

'%(asctime)s.%(msecs)03d %(levelname)-8s %(module)-16s %(message)s'

This translates into the following columns: Time | Information Level | Module logging | Log message (please note that prompted log only includes the Log message).

There are four information levels:

  1. INFO : Log information of all kinds, nothing to worry about.

  2. WARNING : A foreseen/predictable action by the pipeline (input processing, present files) nothing to worry about but sometimes worth checking.

  3. ERROR : Something which should not have happened and that the developer might want to look into but that did not interrupt the execution.

  4. CRITICAL : Something bad which interrupted the execution and that will most probably translate into human intervention either from the user or a bug-fix from the developer.

What follows is a log example:

2019-10-09T03:09:52.876 INFO director ================================================================================
2019-10-09T03:09:52.877 INFO director ADCSng v0.13.2 for Solar_Orbiter run on ESAC01102875 at 2019-10-09 03:09:52.8
2019-10-09T03:09:52.877 INFO director ================================================================================
2019-10-09T03:09:52.877 INFO director
2019-10-09T03:09:52.917 WARNING director MK solo_ANC_soc-pred-mk_V101_20191009_001.tm has not been updated
2019-10-09T03:09:52.955 WARNING director MK solo_ANC_soc-flown-mk_V101_20191009_001.tm has not been updated
2019-10-09T03:09:52.957 INFO director - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2019-10-09T03:09:52.957 INFO director No kernels have been generated.

Sending e-mails

If properly configured ADCSng will send e-mail notifications to users and developers. In order to be able to use it Python needs to connect to the following SMTP server: smtp.sciops.esa.int, the user name and the password are hard-coded in ADCSng and the sender will be esa_spice@sciops.esa.int, so far allowing for an alternative server is not considered. ADCSng will then send notifications in the following cases:

  1. To the specified users after a successful (or partially successful) run, the e-mail template is available here: adcsng/adcsng/etc/email.commnt

  2. To the developer after a successful run with the log of the run.

  3. To the developer in case of a failed run with the log (if applicable) and the Python standard error output (this one is important to be able to detect glitches in the pipeline).

  4. To the developer in case of a lock file present in the directory (more information below).

Exception and Error handling

ADCSng has a limited but concise amount of exception and error handling taking advantages of the possibilities of Python and limited by the error handling of CSPICE (C version of SPICE). Wherever possible ADCSng will provide enough information for the developer to fix the issue.

The ADCSng lock file

When ADCSng is executed, ADCSng works/modifies the source and output directories. Due to this parallel runs of ADCSng on the same directory structure could lead into disaster. ADCSng is triggered by a daemon (crontab) in the SPICE operational server and although in principle executions are planned in such a way this event does not happen, if multiple long orbit files were to be processed, executions might overlap.

The solution has been to implement a locking mechanism. When ADCSng is executed, it will generate an empty lock file as per configuration. If no lock file is provided in the configuration, then the default will be to write it in the directory where ADCSng is being executed with the following name: adcsng_mission.lock

The lock file will be automatically removed at the end of an execution even if it is a failed execution. If a lock file is present when a new execution is requested, ADCSng will not run and if emails are enabled, it will send an e-mail to the developer indicating so.